Designing the Dynamics of Spiking Neural Networks 
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Precise timing of spikes and temporal locking are key elements of neural computation. Here we 
demonstrate how even strongly heterogeneous, deterministic neural networks with delayed interac- 
tions and complex topology can exhibit periodic patterns of spikes that are precisely timed. We 
develop an analytical method to find the set of all networks exhibiting a predefined pattern dynam- 
ics. Such patterns may be arbitrarily long and of complicated temporal structure. We point out 
that the same pattern can exist in very different networks and have different stability properties. 



PACS numbers: 87.18. Sn, 



|.75.Fb, 89. 75. He, 05.45.-a 



u 



(N 
> 

m 
o 
o 

o 
\o 
o 



I , 



X 



Repeated patterns of spikes with temporal precision in 
the millisecond range have been experimentally observed 
in different neuronal systems Q, Q IH They corre- 
late with internal and external stimuli and are thus dis- 
cussed to be essential for neural information processing 
(see, e.g., Q). Their dynamical origin, however, is un- 
known. One possible explanation for their occurrence is 
the existence of excitatorilycoupled feed-forward struc- 
tures, synfire chains 0,0)01) which are embedded in a 
network of otherwise random connectivity and receive a 
large number of random external inputs. Other studies 
point out that spike patterns can originate as attractors 
of deterministic recurrent networks if inhibitory interac- 
tions dominate These studies could already treat 
networks of complicated connectivities and successfully 
found one specific network solution for a given pattern. 
Yet it is still unclear which set of networks have the po- 
tential to realize a given spiking dynamics. Moreover, 
the recent studies considered interactions without delays. 
Delays, however, are known to be significant in biolog- 
ical neural systems 0| and to have a strong impact 
onto even the qualitative dynamics of neural networks 

(cf. E El El EE El 00). 

It is thus still an open question whether and how a 
deterministic network, despite simultaneously exhibiting 
delayed interactions and strong heterogeneities, can yet 
display precisely timed spiking dynamics. If so, what are 
the possible networks that generate a given dynamics? 

In this Letter we study a class of spiking neural net- 
work models with delayed interactions. We provide a 
solution to an inverse problem for networks of arbitrary 
connectivity: We present an exact analytical method to 
find the set of all networks, by determining the coupHng 
strengths, such that they exhibit a given periodic spike 
pattern of arbitrary temporal extent. The analysis shows 
that even arbitrarily large networks with complicated 
connection topologies and strong heterogeneities can yet 
display patterns of spikes that are timed precisely. The 
class of networks realizing a simple periodic pattern, i.e. 



one in which each neuron fires exactly once before the 
sequence repeats, is derived and parameterized analyti- 
cally. The network may have a mixture of both excitatory 
and inhibitory couplings, with the stability of a pattern 
depending on the particular coupHng architecture. 

Consider a network of TV € N oscillatory neurons that 
interact by sending and receiving spikes via directed de- 
layed connections. One phase-like variable </>;(*) specifies 
the state of each neuron I G {1,...,N} at time t. A 
strictly monotonic increasing rise function Ui defines the 
membrane potential Ui(<i>i) of the neuron, representing 
its subthreshold dynamics In the absence of interac- 
tions, the phases increase uniformly obeying d(f)i/dt = 1. 
When 4>i reaches its threshold, (jiiit^) = O/, it is reset, 
4>i(t) = 0, and a spike is emitted. After a delay time 
Tmi this spike signal reaches the post-synaptic neuron to, 
inducing an instantaneous phase jump 
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mediated by the transfer function He"^\(j)) = 
Um^[Um[(f))+s) that is strictly monotonic increasing both 
as a function of e and of 4>. Here, Emi denotes the strength 
of the coupling from neuron Itom. Sending and receiving 
of spikes are the only nonlinear events occurring in these 
systems. For simplicity of presentation, we here focus on 
non-degenerate events: We consider arbitrary periodic 
patterns in which (i) all spikes are sent at non-identical 
times and (ii) received at non-identical times, and (iii) 
neurons receiving a spike do not generate a new spike at 
the same time. We focus on networks of identical neurons 
Ui{4>) = U {4>) with the same intrinsic inter-spike intervals 
fixed by 8; = 1, on identical delays Tmi = t, and pat- 
terns without silent neurons (that do not spike within a 
pattern due to sufficiently strong inibitory input). Below, 
we will explain the underlying ideas of how to find the set 
of all networks exhibiting a given pattern as an invariant 
solution for this class of systems. Nevertheless, based on 
the analysis presented here, the developed method can be 
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FIG. 1: (color) Complicated spike pattern in a small network 
(A'' = 15). (a) Network of eight integrate-and-fire (green) 
and seven MiroUo-Strogatz (blue) neurons with distributed 
thresholds 9; G [0.5,2.0] and delays Tmi G [0.1,0.9]. Each di- 
rected connection between any two neurons is randomly cho- 
sen to be present with probability p = 0.6. Connections are 
either excitatory (black) or inhibitory (red) (thicknesses pro- 
portional to coupling strengths), (b) The spiking dynamics 
(green and blue bars according to neuron type) of the network 
shown in (a) perfectly agrees with the predefined pattern (pe- 
riod T = 1.3) of precisely timed spikes (black bars underlying 
the colored ones). The pattern includes several simultane- 
ous spikes. Three neurons, I G {4, 11, 12}, are switched off 
(non-spiking). 



further extended to cover also different types of neu- 
rons, heterogeneously distributed delays and thresholds, 
and complicated stored patterns that include degenerate 
spikes, multiple firings of the same neuron and silent neu- 
rons that never fire. Figure illustrates such a general 
case. 

What characterizes a periodic pattern of precisely 
timed spikes? Let i^, i G Z, be an ordered list of times 
at which a neuron emits the ith spike occurring in the 
network, such that tj > U if j > i. Assume a periodic 
pattern consists of M spikes. Such a pattern is then 
characterized by its period T, by the times U G [0, T) of 
spikes i € {1, M}, and by the indices Si G {1, . . . , N} 
identifying the neuron that spikes at U . To exclude 
technicalities in the presentation, we assume that for all 
pairs ti and tj of subsequent spike times of each neu- 
ron I, it receives at least one spike within the interval 
{ti,tj) n {ti, ti -1-6;). Periodicity entails ti + nT = ti^nM 
and Si = Si+nM for all n & Z. This imposes condi- 
tions on the time evolution of the neurons' phases. Sup- 
pose a specific neuron / fires at K{1) times ti^ G [0,r), 
k G {1, K{1)} within the first period. For the non- 
degenerate patterns considered, this implies 

0;(trj = i, (2) 

whereas at any other time f G [0,T), t ^ ti^^ for all fc, 

un < 1, (3) 

to prevent untimely firing. The monotonicity of the 



transfer function implies that t he p eriodicity of the pat- 
tern is necessary and sufficient for the periodicity of 
the phases, 

Mt) = Mt + nT), (4) 
for all n G Z and all t G [0, T). We therefore equivalently 
consider 4'i{t) for t G [0,T) with periodic boundary con- 
ditions. All times are measured modulo T and spike time 
labels i are reduced to {1, ...,M} by subtracting a suit- 
able integer multiple of M. Let P{i) G {1, M} denote 
the spike arriving last before the firing time ti such that 
P{i) = aTgmm{t^-9j \j G {1, ...,M}}, where Oj ^tj+r 
is the arrival time of the spike labeled j. 
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FIG. 2: (color) Restriction of a neuron's dynamics between its 
firing events Q. In this example, two spikes arrive between 
the firing times ti and tj of neuron I. The solid line indicates 
one possible time evolution of the phase 4>i{t). Between the 
firing times, (j}i(t) may follow any path within a possibly semi- 
infinite polygon (gray shaded; green dashed lines show other 
possible trajectories). A too large phase at Sp(i)_|.i contradicts 
and will lead to early firing (dark red dashed line). The 
phase at Sp(j) is fixed (red dot). Any other phase inconsistent 
with the equality in (0 would lead to a firing time earlier or 
later than predefined (light red dashed lines). 

Moreover, let Aj = 9j+i — 9j be the time differences 
between two successive arrivals. We can now rewrite 
Eqs. 101 and © for neuron I as a set of conditions on 
the phases 4>i{9i) at each spike arrival time 9i, in terms 
of the firing times ti^ of that neuron and spike arrival 
times 9 
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(t>i{9j) <1- A 



(5) 
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where k G {l,...,K{l)) and j G {1,...,M}, j ^ P{ik) 
for all k. The coupling strengths 1,1' G {!,... ,N} 
of a network realizing a given pattern are now restricted 
by a system of Y^f^i K{1) = M nonlinear equations and 
^j^-^(M — K{1)) — [N — 1)M inequalities originating 
from @ and After a firing of neuron I at time ti 
where its phase is zero, conditions (HJ and © impose 
restrictions at each spike arrival time while the time evo- 
lution proceeds towards the subsequent firing time tj of 
neuron I, as illustrated in Fig. [2 As a result, we have 
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FIG. 3: (color) Two different networks (a), (c) realize the 
same predefined pattern ((b), (d) grey lines). A small ran- 
dom perturbation is applied at the beginning of the second 
period. The network dynamics (spike times relative to the 
spikes of neuron 1 = 1, color coded for each neuron), found by 
exact numerical integration [2^ shows that in network (a) the 
pattern is stable and thus regained after a few periods (b); in 
network (c) it is unstable (d) and eventually another pattern 
will be assumed. 

A particular solution [i^l to the system ijjjl, provides the 
coupHng strengths £;p, V G {1, . . . , N}, of incoming con- 
nections to neuron I. Solutions to systems analogous to 
for all neurons I define the coupling architecture of the 
entire network. Often JJjl is an under-determined system 
such that many solutions exist, implying that many dif- 
ferent networks realize the same predefined pattern; cf. 
Fig. El 

We can then require additional properties from the net- 
work. For instance, a connection from a neuron I to m 
can be absent (requiring the coupHng strength e-mi = 0), 
taken to be inhibitory {emi < 0), excitatory {emi > 0) 
or to lie within an interval. In particular, we can spec- 
ify inhibitory and excitatory subpopulations. In certain 
cases j_such as for networks of leaky integrate and fire neu- 
rons 0, U{cj)) = U^{(j}) = [l-exp(-7</>)]/[l-exp(-7)], 
7 > 0, or Mirollo-Strogatz neurons U{4>) = Ub{4>) = 



6"^ln(l + (exp(6) - l)cj)), 6 > 0, a solution of Q can 
be found in a simple way, because the system is then re- 
ducible to be linear in the couplings or polynomial in its 
exponentials. 

Networks realizing a given pattern do not always ex- 
ist. This can already be observed from a simple exam- 
ple: Consider a pattern with no spike arrival between 
two spikes sent by the same neuron. Due to its free evo- 
lution between the spiking times, their time difference 
must equal the free period; hence a predefined pattern 
with different inter-spike interval is not realizable by any 
network. 

For a simple periodic pattern, the system 10 is guar- 
anteed to have a solution, as long as basic requirements 
(e.g., the delays being smaller than the neurons' intrinsic 
inter-spike-intervals) are obeyed. Without loss of gener- 
ality the neuron firing at time U is labeled I, i.e. si = I 
for / G {1, ...,M = N}. An analytic parameterization of 
all networks realizing such a pattern is then given by 

£lP{l) + l =H^^\g^^^^^^){0Pil) + l -tl), 

£lP{l)+k ^H^\g^^^^^^){(l>l{Op{l) + k-l) + Ap(,) + fe_i), 

£;p(0 =^r_\t,_ep,,j)(0;(^'p(()-i) + Ap(Q_i), (8) 

in terms of the neurons' phases 4'i{9i), l,i G {l,...,N} 
at the spike arrival times. Here k G {2, ...,N — 1} and 
H^^{(j)) is the inverse of He{<j)) with respect to e. The 
phases (j)i{9i) are subject to the restrictions (EJ. This 
parameterization shows that an N{N — l)-dimensional 
submanifold of networks realizing the pattern exists in 
Eij-space. Additional features entail additional condi- 
tions on the phases at the spike arrival times: For in- 
stance, exclusion of self-interaction is guaranteed by the 
conditions 0/(0;) — t if there is no spike- arrival in {ti,9i), 
and 4>i{9i) — 4>i{9i^i) — A/_i otherwise, reducing the di- 
mension of the submanifold of possible networks by N . 
Moreover, requiring the couplings to be purely inhibitory 
leads to the accessibility conditions 

4'i{0p(i)+i) <Op(i)+i -tu (9) 

MO,+i)-HOj)<^j, (10) 

where j ^ P{1)- We can therefore successively choose 
4>i[Sp(i)+r,i), m G {!,..., TV - 1}, starting with m = 1. 
Inequalities ^ and ifTflll hold with reversed relations for 
purely excitatory coupling. Purely inhibitory realizations 
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exist if a pattern has period T > 1 (larger than the neu- 
ron's intrinsic inter-spike interval); otherwise 4>i{tY) = 1 
is not accessible from 4>i{ti) = 0. Similarly, purely exci- 
tatory realizations exist if a pattern has period T < 1. 

Is a pattern emerging in a heterogeneous network sta- 
ble or unstable? We numerically investigated patterns 
in a variety of networks and found that in general the 
stability properties of a pattern depend on the details of 
the network it is reaHzed in, see Fig. for an illustra- 
tion. Depending on the network architecture, the same 
pattern can be exponentially stable or unstable, or ex- 
hibit oscillatory stable or unstable dynamics. For any 
specific pattern in any specific network, the linear stabil- 
ity properties can also be determined analytically, similar 
to the exact perturbation analyses for much simpler dy- 
namics in more homogeneous networks [14, 16). More 
generally, in every network of neurons with congeneri- 
cally curved rise functions and with purely inhibitory (or 
purely excitatory) coupling, a nonlinear stability analy- 
sis shows that the possible non-degenerate patterns are 
either all stable or all unstable. For instance, in purely 
inhibitory networks of neurons with rise functions of neg- 
ative curvature, such as integrate-and-fire neurons, every 
periodic non-degenerate spike pattern, no matter how 
complicated, is stable. 

In summary, we presented a method to find the set 
of all networks reaHzing a predefined periodic pattern 
of spikes, and for imposing additional constraints, for 
instance specifying absent connections and choosing in- 
hibitory or excitatory subpopulations. A predefined sim- 
ple periodic pattern is particularly interesting because a 
network realizing it is guaranteed to exist; here we pa- 
rameterized analytically all such networks. 

In general, these results demonstrate that precise, re- 
producible dynamics arises even in high-dimensional het- 
erogeneous complex systems where it might be unex- 
pected. The design method of solving an inverse problem 
presented here, capable of finding all networks that ex- 
hibit a predefined dynamics, might thus be of interest in 
the theory of coupled oscillators and complex networks 
and contributes a novel perspective to theoretical neuro- 
science. The method can be extended to include hetero- 
geneities in all parameters and non-degenerate patterns 
hidden neurons outside a core network, as well as 
non-periodic patterns, by dropping the periodicity con- 
straint 0. 

In particular, our results shed a new light on how pat- 
terns of precisely timed spikes may emerge in determin- 
istic neural network dynamical systems, even for biolog- 
ically realistic architecture. For instance these networks 
may simultaneously exhibit strongly heterogeneous pa- 
rameters, compHcated topology, and substantial and dis- 
tributed delays. However, future work still needs to fully 
answer how experimentally observed synchronization j2| 
as well as recurrent patterns of spikes []J,y,|j| really arise. 
Important topics of theoretical research include, for in- 



stance, (i) the exact dynamics of networks of excitable 
neurons that are either excited by recurrent network in- 
puts or by external stimuli [i^l, (ii) spike patterns that 
are not periodic but separated by intervals of irregular 
activity j2J|, as well as (iii) cortical songs which are se- 
quences of patterns occurring repeatedly in the same or- 
der but varying in the timing between patterns [J Ql- 
Studies in these directions could further clarify mecha- 
nisms used for information processing in such networks. 
In particular, it will be exciting to see whether biological 
neural networks rely on stochastic features of topology 
and input Q or the exact wiring diagram and precise 
single neuron dynamics play a significant role in creating 
a temporal code. 

We thank S. Jahnke, F. Wolf and T. Geisel for useful 
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under grant number 01GQ0430. 
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